Drive-by calibration from static targets

ABSTRACT

A method for deriving extrinsic camera parameters of a vehicle camera. Calibration markers are provided on a flat ground surface and the vehicle is driven past the calibration markers. Marker boundaries are detected and matched to stored pre-determined shape parameters and a marker shape is identified. At least one extrinsic parameter of the camera is derived using the tracked positions of identified marker shape in the video sequence captured while vehicle is moving, wherein the extrinsic parameter is selected from mounting positions and a rotation is selected from both horizontal axis and vertical axis of a vehicle coordinate system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to European Patent Application No.EP15178814, filed Jul. 29, 2015, the contents of such application beingincorporated by reference herein.

FIELD OF THE INVENTION

The present specification relates to a camera calibration of a vehiclecamera and in particular to a camera calibration using static targets onthe ground plane.

BACKGROUND OF THE INVENTION

Increasingly, passenger vehicle are equipped with cameras. The camerascan provide many assistance functions to the driver such as lanekeeping, parking aid, detection of obstacles and other objects,emergency braking, reconstruction of accidents and other events,monitoring of the traffic and weather situation, recording of drivingpatterns, deriving speed and position of the car, vehicle detection andcollision avoidance, detection of traffic signs, lighting adjustment,night vision and more.

These functions require image processing, which can be provided byprocessing units in the cameras or dedicated processing units in thecar. The processing units are connected to further devices, such asscreens, indicators, the engine control unit, etc.

SUMMARY OF THE INVENTION

For an accurate representation of the vehicle surroundings, a cameracalibration is required. The camera calibration comprises thecalibration of intrinsic parameters, such as focal length and lensdistortion, and of extrinsic parameters such as camera placement andorientation.

In particular, an aspect of the present specification discloses a methodfor an extrinsic calibration of a single camera or of a multi-camerasystem according to the present specification. A calibration methodaccording to the application does not require

-   -   a precise placement of the vehicle    -   a dedicated static vehicle time for calibration    -   a large surface area within a factory, although these conditions        may be provided, if necessary.

The calibration method can be performed inside or outside the factory.For example, the calibration may be performed in the garage of aworkshop or on a dedicated place, which has appropriate markings on aground surface.

A calibration method according to an aspect of the present specificationcan provide

-   -   a short calibration time    -   flexible driving condition requirements    -   independence of the ground surface as long as the surface is        flat enough, while providing a sufficiently accurate        calibration.

In a method according to an aspect of the present specification, avehicle is driven over a set of predetermined patterns or markers on theground. Cameras, which are mounted on the vehicle, record images ofthose patterns. One or more computation units calculate the appropriateextrinsic parameters for all cameras.

The extrinsic parameters relate the camera coordinate system to a worldcoordinate system. This can be done by a rotation and a translation, forexample.

The patterns, which are also referred to as “calibration targets” or“calibration markers”, are not necessarily aligned parallel to eachother. When the vehicle is driving towards the targets, one or moretargets are detected, identified and tracked in the captured videosequence. The calculation of extrinsic camera parameters is based onmultiple frames of the captured video sequence.

According to a main embodiment, extrinsic camera parameters, whichinclude the rotations about the x, y and z axis of a vehicle coordinatesystem and the mounting height in the vehicle coordinate system areoptimised at the same time and the vehicle cameras are calibratedindividually. In particular, there may be four vehicle cameras.

The properties of a calibration method according to the presentspecification can also be summarized according to the following lists:

-   -   the calibration patterns do not need to be aligned,    -   no other information from the vehicle is needed such as speed,        steering, GPS location . . .    -   intrinsic parameters are not calculated, in fact they are one of        the inputs of the method.

1. The marker shapes are pre-determined, and are already storedaccording to the method. Only the following input is required:

-   -   the input from the camera(s) (sequence of Images)    -   the camera intrinsics (which are a given to the camera when it        is manufactured)    -   the estimated camera mounting position (their placement on the        vehicle, in particular, the X and Y positions)

2. A sequence of images from the camera on a moving vehicle is used.

3. The patterns are not provided by road markings but by predefinedspecific patterns.

4. a preferred method for providing the patterns is painting thepatterns (for fixed usage like in a factory) or having them pre-printedon boards that are portable (for more flexibility, for example for usein garages).

The computation unit makes use of a known true size of the patterns,wherein the size refers to the outer boundaries of the patterns.Furthermore, the patterns may comprise a sub-structure or sub-pattern,such as a checkerboard pattern.

In particular, the present application discloses a method for derivingextrinsic camera parameters of at least one vehicle camera.

One or more pre-determined calibration markers are provided on a flatground surface. A pre-determined set of shape parameters, whichcorrespond to the calibration markers is stored in a computer readablememory of the vehicle. In a broader sense, a stored vector graphic ofthe marker is also regarded as shape parameters. The shape parameterscharacterize the calibration markers. For example, a radius cancharacterize a circular marker and a cross diagonal length or a sidelength can characterize a square shaped marker.

The vehicle is driven past the one or more pre-determined calibrationmarkers and the calibration markers are captured by the vehicle cameraor by the vehicle cameras. According to another embodiment, the vehicleremains stationary while the images are captured. Image data is receivedfrom the at least one vehicle camera and processed in a computation unitof the camera and/or in a computation unit of the vehicle. The imagedata that is relevant for the present method comprises calibrationmarker images. The marker images are digital representations of a sensorsignal corresponding to the marker or to a portion of the marker.

The portion of the image data, which comprises calibration marker imagesis also referred to as “calibration marker image data”. The expressions“image data”, “calibration marker image” and “calibration marker imagedata” may refer to the digitized raw image data of the sensor or toprocessed data. Among others, the image processing may include areduction of distortions, providing a top-down projection, a noisereduction, or a compensation for properties of the sensor elements.

The computation unit detects marker boundaries of the one or morecalibration marker imagers in image frames of the image data and matchesthe detected marker boundaries to the pre-determined shape parameters.The computation unit identifies one or more calibration markers. In onerealization, the markers are identified by identifying the shapes of thecalibration markers. The shapes of the markers are identified bycomparing the apparent shape of the calibration markers with the storedshape parameters

The identified calibration markers in the video sequence are trackedwhile the at least one vehicle camera is moving, and at least oneextrinsic parameter of the at least one vehicle camera is determinedusing the positions of the identified calibration markers in thecaptured video sequence.

In one embodiment, the calibration markers are provided with an opticalrecognition symbol, which identifies the type of the marker. Acomputation unit can then already identify the marker shape byidentifying the recognition symbol. Among others, the opticalrecognition symbols may differ in colour, number or arrangement. Forexample, a square marker may be provided with a red dot and arhombohedric marker with a blue dot or they may be provided withdifferent numbers and/or positions of dots. Alternatively or inaddition, the marker shape can be identified by the outer makerboundary, a marker pattern or by other characteristic optical features.

In one embodiment, the optical recognition symbol also allows to detectthe alignment of the marker, for example by providing a cross shaped oran arrow shaped optical recognition symbol on the marker or by providingan internal pattern, such as a checker board pattern.

Specifically, the derived extrinsic parameters comprise a rotation ortilt around a first horizontal axis or x-axis of a vehicle coordinatesystem, also known as pitch rotation, a rotation around a secondhorizontal axis or y-axis, also known as a roll rotation, a rotationaround a vertical axis or z-axis, also known as yaw rotation, and amounting height with respect to the vehicle coordinate system.

The vehicle coordinate system refers to a fictitious rectangularcoordinate system or frame of reference, which is fixed to the vehicle.The horizontal axes are parallel to the ground plane and aligned alongthe principal axes of the car while the vertical axis points upward andis orthogonal to the horizontal axes.

According to one choice of vehicle coordinate system, the horizontalplane of the vehicle coordinate system is aligned with the ground plane,the x-axis is parallel to the front axis and the coordinate original isat a left end of the front axis. According to another choice, thecoordinate origin is the projection of the center of mass to the groundplane or the projection of a geometrical centre point to the groundplane.

In one particular embodiment, the method comprises identifying a firstset of at least two parallel straight marker boundaries in the image ofone of the one or more pre-determined calibration markers andidentifying a second set of at least two parallel straight markerboundaries in the image of one of the one or more pre-determinedcalibration markers. The marker boundaries are selected such that thesecond set of marker boundaries is not parallel to the first set ofmarker boundaries. The second set of marker boundaries and maycorrespond to an image of the same or of a different marker.

The computation unit derives an orientation of the at least one vehiclecamera with respect to the ground plane from the first set and thesecond set of marker boundaries.

The two sets of parallel lines converge on two vanishing points whichare on the horizon, which is the vanishing line connecting the twovanishing points. This in turn provides the tilt of the camera withrespect to the ground plane. The tilt provides the camera rotationsabout the horizontal x and y axes of the vehicle coordinate system.

According to another embodiment, a camera tilt against the vertical isdetermined from the perspective foreshortening of the image of themarkers on the sensor surface of the vehicle camera. According to thisembodiment, a perspective foreshortening of an image of at least onemarker is derived from the matching of the detected marker boundaries tothe pre-determined shape parameters and an orientation of the at leastone vehicle camera with respect to the ground plane is derived based onthe derived perspective foreshortening.

By way of example, a marker image of a circular marker appears as anoval. The degree of ellipticity and the direction of the principal axesof the ellipse give the amount and the direction of tilt. For atriangularly shaped marker, the three angles can be compared with thecorresponding angles of the triangular marker, which are stored as shapeparameters in the computer readable memory. Thereby, a tilt of thecamera is derived. The same or a similar method can also be applied todetermine a perspective foreshortening of a polygon shaped marker. Byway of example, the polygons may be decomposed into triangles and theangles of the triangles or the angles between the outer markerboundaries can be compared to corresponding stored angles of thecalibration markers.

In order to increase the accuracy, this estimate can be repeated fordifferent images of calibration markers and/or for different imageframes of an image sequence and the estimate can be obtained as aweighted average of the individual estimates. If an accuracy of anestimate is considered insufficient, it can be given a weight of zero inthe weighted average.

In one embodiment, the marker boundaries comprise a boundary of aninternal pattern of one of the one or more pre-determined calibrationmarkers, in another embodiment, the marker boundaries comprise an outerboundary.

In particular, the sets of parallel marker boundaries or the perspectiveforeshortening can be used to determine a rotation around the first orthe second horizontal axis from the derived orientation of the camerawith respect to the ground plane.

According to a further embodiment, the method also comprises deriving amagnitude of a shape parameter in a calibration marker image at leastone of the one or more calibration markers, such as an apparent borderlength, an apparent cross diagonal length, a surface area, or a lengthof a principal axis. The magnitude of the shape parameter, whichprovides a measure for the apparent size of the marker, is compared witha magnitude of a corresponding stored shape parameter, and a mountingheight of the at least one camera is derived from the comparison of themagnitudes.

In particular, the magnitude of the shape parameter, or the apparentsize of the marker in the calibration marker image allows determining adistance d of the marker from the vehicle camera. When the tilt a of thecamera against the vertical is known, for example by using theabovementioned method or otherwise, then the mounting height Z of thecamera with respect to the ground surface can be derived as Z=d*cos (α).

According to a further embodiment, the method comprises deriving amagnitude of a shape parameter of an image of one of the one or morecalibration markers from image data, or from an image frame, of a firstvehicle camera. The size parameter is indicative of the size of thecalibration marker image on the image sensor.

By comparing the size parameter with a stored size parameter, acomputation unit derives a distance between the first vehicle camera andthe corresponding calibration marker, which corresponds to thecalibration marker image. Furthermore, the computation unit derives amagnitude of a shape parameter of a calibration marker image, whichcorresponds to the same calibration marker, from image data or from animage frame, of a second vehicle camera. By comparing the size parameterwith a stored size parameter, a distance between the second vehiclecamera and the corresponding calibration marker is derived.

By using the first distance and the second distance a position of thecalibration marker on the ground surface with respect to a vehiclecoordinate system is derived.

A rotation of the first vehicle camera around the vertical axis isderived from the derived position of the calibration marker.Furthermore, a rotation of the second vehicle camera around the verticalaxis is derived from the derived position of the calibration marker.

According to a further modification of the method, the method comprisesderiving a rotation of the first or of the second vehicle camera aroundthe first horizontal axis or around the second horizontal axis by usingthe derived position of the corresponding calibration marker.

The accuracy may be improved by computing a weighted average of theestimates of the external parameters. By way of example, the image datamay comprise two or more calibration marker images. The computation unitderives a camera orientation of the at least one camera for each of thetwo or more calibration marker images and computes an average cameraorientation as a weighted average of the derived camera orientations ofthe at least one camera.

The calibration markers may be provided in various ways. In particular,they can be provided by painted surfaces or by one or more portableboards.

In a further aspect, the current specification discloses a computerprogram product for the execution of the abovementioned method, whichmay be provided on a storage medium such as an optical medium, amagnetized medium, a ferroelectric medium, or in a memory of anintegrated circuit/ASIC, on an EPROM etc. Furthermore, the currentspecification comprises a computer readable storage medium, whichcomprises the computer program product.

In a further aspect, the current specification discloses a computationunit, such as one or more computation units of vehicle cameras, acentralized computation unit for connection to the surround view camerasor a combination thereof. In particular, the vehicle cameras can be partof a surround view system, which provides a top down projection of astreet surface.

The computation unit comprises an electronic computing means such as amicroprocessor, an IC or and ASIC, a computer readable memory withpre-determined shape parameters of a set of pre-determined calibrationmarkers, and a connection for receiving image data from one or morevehicle cameras. The image data comprises calibration marker image data.

By providing stored executable instructions and/or correspondingelectronic components for digital and/or for analog data processing thecomputation unit is operative to detect marker boundaries in one or morecalibration marker images from the image data and match the detectedmarker boundaries to the pre-determined shape parameters.

The computation unit tracks the one or more identified calibrationmarkers in the video sequence while the at least one vehicle camera ismoving and derives at least one extrinsic parameter of the at least onevehicle camera using the positions of the one or more identifiedcalibration markers in the captured video sequence.

Among other, the extrinsic camera parameter can be selected from arotation around a first horizontal axis, a rotation around a secondhorizontal axis, a rotation around a vertical axis, or Z-axis, of avehicle coordinate system, and a mounting height with respect to thevehicle coordinate system.

In a further embodiment, the computation unit is furthermore operativeto derive a relative orientation of at least one vehicle camera withrespect to an orientation of corresponding calibration markers on aground surface, which correspond to the calibration marker images.

Furthermore, the current specification discloses a kit with thecomputation unit and one or more vehicle cameras, and a vehicle in whichthe kit is installed. The one or more vehicle cameras are mounted topre-determined positions of the vehicle and are facing to the exteriorof the vehicle. The computation unit is electrically connected to theone or more vehicle cameras.

The markers do not have to be aligned in a pre-determined orientation orlayout pattern with respect to each other, although they may have apre-determined relative orientation or layout pattern, which can be usedin a feature detection and matching.

Due to perspective foreshortening some marker boundaries are projectedto fewer image pixels than others and are therefore less suitable toderive an orientation of the marker boundary. When the markers are laidout in random orientation, it may result in a higher likelihood to finda marker boundary, which has a suitable orientation with respect to avehicle camera. Moreover, the patterns can be laid out conveniently andquickly when the relative distances and orientations do not need to befixed in advance. This feature can be advantageous in the garage of aworkshop and can help the motor mechanics to save time.

When the vehicle drives past the markers, the subsequent image framesprovide different view of the markers. This also increases thelikelihood that a particular view of a given marker of a particularimage frame is suitable for determining a camera orientation or mountingheight.

Differently shaped markers can be used to enhance the likelihood to findmarker boundaries that are suitable for image recognition. For example,the markers may vary with respect the number of straight outerboundaries, the length of the boundaries and the surface area. Somemarkers may be more suitable for determining a size of the marker.Marker shapes include, among others, a square shape, a rectangularshape, a rhombohedral shape, a triangular shape, an equilateral triangleshape, a right triangle shape, and a circular shape. Regular polygons orpolyhedral with more than four sides may also be used as marker shapes.

The marker boundaries can be used for determining a vanishing line andthereby deriving a tilt of the vehicle cameras with respect to theground plane. Since all markers are arranged on the ground plane, theextension of the marker boundaries end in vanishing points, which areapproximately on the same vanishing line.

According to further embodiments the markers are provided in uniformwhite colour, or in any other uniform colour, and the patterns couldcomprise colours other than white or black. The markers 21 to 27 can beprovided as pre-defined markers on any flat surface or a dedicatedcalibration surface. In particular, the markers 21 to 27 can be providedby painting, for example for a factory calibration, or as pre-printedpattern portable boards, for example for a calibration in a workshopgarage.

According to one embodiment, an image evaluation unit estimates thedimensions of the markers using information such as the distances of thecameras to the ground plane, the orientation of the cameras relative toa vertical axis, the orientation of the car relative to the markers inthe horizontal x-y plane and the speed of the car.

If the markers are provided as customized markers, the markers may beprovided as physical markers on the ground surface, such as: usingtemporary portable patterns or disposable stickers or by painting themon the road surface such as using paint or paint with reflectingparticles or components.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject of the present specification is now explained in furtherdetail with respect to the following Figures in which

FIG. 1 shows a vehicle with a surround view system driving past a firstset of markers on a ground plane,

FIG. 2 shows the set of markers of FIG. 1 while the vehicle of FIG. 1 isin a second position,

FIG. 3 shows the set of markers of FIG. 1 while the vehicle of FIG. 1 isin a third position,

FIG. 4 shows a vehicle with a surround view system driving past a secondset of markers,

FIG. 5 illustrates a projection to the ground plane,

FIG. 6 illustrates in further detail the projection to the ground plane,

FIG. 7 shows a procedure for determining external camera parametersusing the markers of FIG. 1 or FIG. 4,

FIG. 8 illustrates a first procedure for determining horizontal cameraorientations,

FIG. 9 illustrates a second procedure for determining horizontal cameraorientations,

FIG. 10 illustrates a procedure for determining a translation of thevehicle,

FIG. 11 illustrates a procedure for determining vertical cameraorientations based on marker sizes, and

FIG. 12 illustrates a procedure for determining horizontal and/orvertical camera orientations based on marker sizes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description, details are provided to describe theembodiments of the present specification. It shall be apparent to oneskilled in the art, however, that the embodiments may be practisedwithout such details.

FIGS. 1 to 3 show the relative position of a set of markers 21-27 to acar 10 while the car 10 drives over the markers and 21-27, and anoptical recognition system of the car takes a video sequence using oneor more of the vehicle cameras 12, 13, 14, 15.

FIG. 1 shows a car 10 with a surround view system 11. The surround viewsystem 11 comprises a front view camera 12, a right side view camera 13,a left side view camera 14 and a rear view camera 15. The cameras 12-15are connected to a CPU of a controller, which is not shown in FIG. 1.

In FIG. 1, a world coordinate system is indicated by coordinates X, Y, Zand a vehicle coordinate system is indicated by coordinate system X′,Y′, Z′. The vehicle coordinate system X′, Y′, Z′ is aligned with thevehicle and is attached to the vehicle, such as the centre of gravity,the centre of the front axis or the projection of those points to theground surface. Likewise, the world coordinate system X, Y, Z isattached to a point in the surroundings, such as for example thelocation and orientation of the vehicle coordinate system at a time t_0.

The orientation and placement of the coordinate system is provided byway of example and may be chosen differently. For example, the Z-axismay also point downwards, as shown in FIG. 4.

FIG. 1 shows a situation in which a car 10 drives past or over a set ofcalibration targets or markers 21 to 27, which are arranged on a groundplane 16. An arrow 5 indicates a heading and velocity of the vehicle 10.In the embodiment of FIG. 1, the markers 21 to 27 are defined by theirboundaries 20. The markers 21 to 25 are filled out in a uniform blackcolour while the markers 26 and 27 comprise black and white patterns.The colour may comprise reflective particles.

As indicated in FIG. 1, the direction 5 in which the car drives does notneed to be aligned with the markers. Even if the relative orientation ofthe car towards the markers is not known, it is still possible todetermine the relative orientation of the four cameras in the horizontalx-y plane and the tilting of the cameras relative to the verticalz-axis.

FIG. 4 shows the vehicle 10 driving past a second calibration pattern onthe ground surface 16. Different from the markers of FIG. 1, the markersof FIG. 4 comprise a first type of markers 28, 29, 31, 32 with acheckerboard calibration pattern and a second type of markers 30 with aradial or star shaped calibration pattern.

FIGS. 5 and 6 illustrate a top-down projection or projection to a groundplane.

FIG. 5 shows a side elevation in which the right camera 13 is used toperform a ground plane projection. If the camera is positioned at aheight H above ground, a projection that extends over an angle θcorresponds to a stretch H*cos (θ) of the ground surface.

FIG. 6 shows a view port plane and a ground plane. FIG. 6 refers to atop-down view, in which the ground plane is parallel to the view portplane. In general, the ground plane can be tilted with respect to aplane of the camera sensors, which is also referred to as image plane.The top-down view of FIG. 6 is obtained by deriving a camera orientationrelative to the vertical axis and rotating the image plane such that itis parallel with the ground plane.

A point in a view port plane 17 is denoted by p=(u, v) and acorresponding point in the ground plane 16 is denoted by P=(X, Y). Adistance between the view port plane 17 and a projection centre C isdenoted by the letter “f”.

In the surround view system of FIG. 1, the top-down views of theindividual cameras are merged to obtain a top-down view of thesurroundings of the car, which is similar to a top-down view that wouldbe generated by a camera that is positioned above the car and alignedparallel to the ground surface.

By way of example, the merging of the images of the individual camerascan be performed by using known fields of view of the cameras and knownor derived orientations of the cameras and/or by identifyingcorresponding features in the images of the individual cameras.

FIG. 7 shows a camera calibration of extrinsic camera parameters with anarrangement of calibration markers according to the presentspecification. FIGS. 1 and 4 provide examples for such arrangements.However, the arrangement of calibration markers is not limited to theembodiments of FIGS. 1 and 4.

An automatic determination of extrinsic camera parameters links a cameracoordinate system to a world coordinate system, by using an arrangementof calibration markers according to the present specification.

In a step 40, image data is acquired from the cameras are processed. Ina step 41, marker boundaries are identified, for example by using peaksof a Hough transform and/or clustering of points on a unit sphere ontowhich points (u, v) of the viewport are projected. The identifiedboundaries of the markers are compared with marker boundaries accordingto stored shape parameters of the pre-defined markers.

The shape parameters of the pre-determined markers are stored in acomputer readable memory, which may be provided, among others, by amemory of the vehicle cameras, as an external memory of the vehiclecameras or as a memory of a computation unit that is connected to thevehicle cameras.

In a step 42, patterns of the markers are identified and, if the markercomprises a pattern, the identified pattern is compared with storedshape parameters of the pre-defined pattern. If a matchingpre-determined marker is found, which corresponds to the markerboundaries and, if present, to the pattern, the corresponding pixels ofthe image are identified as pixels of the pre-determined marker.

In a step 43, the mounting heights Z and the vertical orientations x, y,z of the cameras are determined using the pre-determined shapes andsizes of the markers and the marker patterns.

FIGS. 8 to 11 provide examples of carrying out the step 42, while FIG.12 provides an example of carrying out step 43.

In FIG. 8, a first dashed line 50 indicates an orientation of a sensorplane of the front camera 12 in the horizontal X-Y plane and a seconddashed line 51 indicates an orientation of a sensor plane of the rightcamera 13 in the horizontal X-Y plane.

By way of example, a connecting line from a front left corner of thepattern 27 to the front camera 12 is inclined by an angle a₂ withrespect to the sensor plane 50 of the front camera and a connecting linefrom the front left corner of the pattern 27 to the right camera 13 isinclined by an angle a₁ with respect to the sensor plane 51 of the rightcamera 13.

According to one embodiment, the relative orientation of the sensorplanes 50, 51 in the horizontal plane is determined by computing thedifference between the angles a₁ and a₂, This estimate can be improvedby using multiple reference points of multiple patterns and computing anaverage, as well as by using reference points from multiple image framesinstead of using reference points from just one image frame.

According to one embodiment, an absolute orientation of the cameras 12,13, 14, 15 in the horizontal X-Y plane is established by comparison witha reference camera, which has a known orientation. The reference cameramay be provided at a location where it is less likely to receive shocksthat can change the camera orientation, such as behind the windscreen ofthe vehicle.

FIG. 9 shows a second method, in which an orientation of straight linesof the patterns 21 to 27 is used to determine an orientation of thecameras 12, 13, 14, 15 in the horizontal X-Y plane. The straight linesmay be provided by the marker boundaries or also by straight lines ofinternal patterns of the markers.

By way of example, a right front boundary of pattern 25 is inclined atan angle γ₂ with respect to the sensor plane 50 of the front camera 12and a front boundary of pattern 27 is inclined at an angle β₂ withrespect to the image sensor plane of the front camera 12.

Furthermore, the front boundary of pattern 27 is inclined at an angle β₁with respect to the image sensor plane of the right camera 13. In theexample of FIG. 4, the markers are aligned in parallel and with apre-determined distance with respect to each other. The pre-determinedparallel alignment and distance can be used to derive the extrinsicparameters of the camera. In this case, the relative orientation of themarkers and/or their relative distances are also stored in the computerreadable memory of a computation unit.

FIG. 10 shows a method of determining a translation 52 of the vehicle10. By way of example, a front surface of the pattern 22 extends over anangle δ in a first position of the vehicle 10 and over an angle δ′ in asecond position of the vehicle 10. A connecting line from a front leftcorner of the pattern 22 to the front camera 12 appears under an angle εin the first position and under an angle ε′ in the second position.

For a straight motion of the vehicle 10, as shown in FIG. 10, the borderlines of the patterns 25, 27 appear under the same angle relative to thesensor planes 50, 51 of the cameras 12, 13.

The derived motion of the vehicle 10 can then be used to improve anestimate of the extrinsic camera parameters and in particular to improvean estimate of the camera orientations.

FIG. 11 illustrates a method for estimating an inclination of thecameras relative to the vertical. A cross diagonal of the pattern 26appears under an angle ω₂, when viewed from the right camera 13, and across diagonal of the pattern 25 appears under an angle ω₁ when viewedfrom the front camera 12. Using the apparent size of the patterndiagonal of the pattern 26, a distance d between the camera 13 and thepattern 26 is derived. By using the known elevation Z of the camera 13,an angle Φ under which the pattern appears against the vertical isderived as Φ=arcsin (Z/d).

The angle Φ is compared with an angle between a connecting line of thepattern to the camera 13 and the sensor plane of the camera 13. Thereby,a tilt of the sensor plane of the right camera 13 against the verticalor against the horizontal is determined. This estimate can be improvedby using boundary lines of multiple markers, by using multiple frames orby using more shape parameters of the markers such as using both crossdiagonals, using internal patterns of the markers etc.

In particular, the tilt of the sensor plane can be obtained by computinga weighted average of the derived tilt angles from the patterns, shapeparameters and patterns of multiple image frames. In one embodiment, theweights of the weighted average are adjusted according to an estimatedaccuracy of the estimates, wherein less accurate estimates have asmaller weight. For example, the accuracy of a tilt angle estimate isgenerally less for an estimate that is based on a marker which ispositioned further away from the camera.

FIG. 12 illustrates a further method of determining horizontal and/orvertical camera orientations based on marker sizes. By way of example,the front boundary of the marker 27 extends over an angle ω₁ when viewedfrom the front camera 12 and extends over and angle ω₂ when viewed fromthe right camera 13. The respective distances d₁ and d₂ of the marker 27from the front camera and the right camera are computed by using theangles ω₁, ω₂ and the stored real size and shape of the markers as inputvalues. Furthermore, the distance d₃ is computed using the known camerapositions on the vehicle.

Thereby, all three angles of the triangle, which is formed by the frontcamera 12, the marker 27 and the right camera 13, are known.Furthermore, the elevations of the front camera 12, the right camera 13and the marker 27 are also known. This in turn allows to determine theorientation of the triangle in three dimensions.

By using the dimensions of the calibration markers and the orientationof the triangle, the horizontal and vertical camera orientations, or theorientations of the respective camera sensor planes, can be derived. Forenhanced accuracy, this procedure can be repeated for multiple markers,multiple frames and for different shape parameters of the individualmarkers, such as the cross diagonals, the front and the rear boundaries,etc.

REFERENCE NUMBER LIST

10 vehicle

11 surround view system

12 front camera

13 right camera

14 left camera

15 rear view camera

16 ground plane

21-27 markers

28-32 markers

40-48 method steps

50 sensor plane

51 sensor plane

52 translation

1. A method for deriving extrinsic camera parameters of at least onevehicle camera, comprising: providing one or more calibration markers ona ground surface, storing pre-determined shape parameters correspondingto the one or more calibration markers in a computer readable memory ofthe vehicle, receiving image data from the at least one vehicle camera,the image data comprising a video sequence, the video sequencecomprising calibration marker image data, detecting marker boundaries inthe image data, matching the detected marker boundaries to thepre-determined shape parameters, identifying one or more calibrationmarkers, tracking the one or more identified calibration markers in thevideo sequence while the at least one vehicle camera is moving, andderiving at least one extrinsic parameter of the at least one vehiclecamera using the positions of the one or more identified calibrationmarkers in the video sequence.
 2. The method according to claim 1,wherein the marker boundaries comprise a boundary of an internal patternof a calibration marker image.
 3. The method according to claim 1,comprising: deriving a perspective foreshortening of a calibrationmarker image from a matching of detected marker boundaries to thepre-determined shape parameters, and deriving an orientation of the atleast one vehicle camera with respect to the ground plane from thederived perspective foreshortening.
 4. The method according to claim 1comprising: deriving an orientation of the at least one vehicle camerawith respect to the ground plane from the at least one extrinsicparameter, and deriving a rotation around a first horizontal axis fromthe derived orientation of the camera with respect to the ground plane.5. The method according to claim 1 comprising: deriving an orientationof the at least one vehicle camera with respect to the ground plane fromthe at least one extrinsic parameter, and deriving a rotation around asecond horizontal axis from the derived orientation of the camera withrespect to the ground plane.
 6. The method according to claim 1comprising: deriving a magnitude of a shape parameter of a calibrationmarker image, comparing the magnitude of the shape parameter with amagnitude of a corresponding stored shape parameter, and deriving amounting height of the at least one camera from the comparison of themagnitudes.
 7. The method according to claim 1 comprising: deriving amagnitude of a shape parameter of a first calibration marker image fromimage data of a first vehicle camera, deriving a distance between thefirst vehicle camera and the corresponding calibration marker, derivinga magnitude of a shape parameter of a second calibration marker imagefrom image data of a second vehicle camera, the first calibration markerimage and the second calibration marker image corresponding to the samecalibration marker, deriving a distance between the second vehiclecamera and the corresponding calibration marker, deriving a position ofthe corresponding calibration marker on the ground surface with respectto a vehicle coordinate system using the first distance and the seconddistance, deriving a rotation of the first vehicle camera around avertical axis from the derived position of the corresponding calibrationmarker, and deriving a rotation of the second vehicle camera around thevertical axis from the derived position of the corresponding calibrationmarker.
 8. The method according to claim 7, comprising deriving arotation of the first vehicle camera around a first horizontal axis fromthe derived position of the corresponding calibration marker.
 9. Themethod according to claim 7, comprising deriving a rotation of the firstvehicle camera around a second horizontal axis from the derived positionof the corresponding calibration marker.
 10. The method according toclaim 1 wherein the image data comprises two or more calibration markerimages, comprising: deriving a camera orientation of the at least onecamera for each of the two or more calibration marker images, andcomputing an average camera orientation as a weighted average of thederived camera orientations of the at least one camera.
 11. The methodaccording to claim 1 wherein the one or more calibration markers areselected from one or more painted surfaces and one or more portableboards.
 12. A non-transitory computer program product for the executionof a method according to claim
 1. 13. A computer readable storagemedium, the computer readable storage medium comprising the computerprogram product according to claim
 12. 14. A computation unit, thecomputation unit comprising: a computer readable memory, the computerreadable memory comprising stored pre-determined shape parameters of aset of calibration markers, and a connection for receiving image datafrom one or more vehicle cameras, the image data comprising a videosequence, the video sequence comprising calibration marker image data,wherein the computation unit is operative to detect marker boundaries inthe image data, match the detected marker boundaries to thepre-determined shape parameters, identify one or more calibrationmarkers, track the one or more identified calibration markers in thevideo sequence while the at least one vehicle camera is moving, andderive at least one extrinsic parameter of the at least one vehiclecamera using the positions of the one or more identified calibrationmarkers in the video sequence.
 15. A kit of the computation unitaccording to claim 14 and one or more vehicle cameras.
 16. A vehiclecomprising the kit according to claim 15, wherein the one or morevehicle cameras are mounted to pre-determined positions of the vehicleand are facing to the exterior of the vehicle, and wherein thecomputation unit is connected to the one or more vehicle cameras. 17.The method according to claim 8, comprising deriving a rotation of thefirst vehicle camera around a second horizontal axis from the derivedposition of the corresponding calibration marker.